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ABSTRACT 



The Department of Measurement, Statistics, and Evaluation 
(EDMS) at the University of Maryland is working to develop Master’s degree 
programs that are oriented around developing assessment professionals for 
work in applied settings. Two fundamentally different sets of experiences are 
being developed: (1) assessment development, administration, and scoring; and 

(2) data analysis and reporting. Organizations interested in -participation 
would be asked to transfer funds for each intern they sponsor and provide 
professional supervision of the intern's experiences. The intern would spend 
approximately 15 hours a week for a significant period of time at the 
organization's job site. The student, who would learn by doing, would 
undertake assessment related duties and then prepare a paper that can be used 
to assess the student's learning. The student may also use the paper to 
fulfill the "Scholarly Paper" requirement of the Master of Arts degree. All 
students would complete a core of courses in assessment and measurement and 
then would choose between two specializations. Competencies are described for 
each specialization and a sample schedule is presented. Prospective 
sponsoring organizations have been interested in employing graduates of a 
program of this type, but many have been reluctant to contribute their sites 
as placements for paid interns. Course developers expect that the interns 
will increase rather than drain the capacity of the organization. (SLD) 
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This paper is prepared for the: 

Annual Meeting of the American Educational Research Association in Montreal Canada 

April 2000 



Educating Applied Assessment Professionals at the Masters Level 

William D. Schafer 
University of Maryland 

We have been developing masters programs in my department (EDMS) at the University 
of Maryland that are oriented around developing assessment professionals for work in 
applied settings. We are planning to implement two fundamentally different sets of 
experiences. 

Assessment Development, Administration, and Scoring 

Data Analysis and Reporting 

We currently have received positive expressions of interest from three organizations to 
fund and supervise one or more students in an internship arrangement. The organization 
would be asked to transfer to the EDMS department $14,500 and to provide professional 
supervision of the intern’s experiences. 

The intern would spend approximately fifteen hours per week for a significant period of 
time engaged in a planned series of activities at the organization’s job site. The activities 
the intern would be asked to undertake would be appropriate for someone employed for 
the relevant position description in that organization, with the degree of responsibility 
and supervision that the organization is comfortable assigning to the individual. Our 
intent is for the student to “learn by doing” as well as to document the knowledge, skills, 
and abilities the student develops. We hope each organization will find it mutually 
beneficial to help us use these programs to train professionals who can fill the sorts of 
assessment-related positions that exist in their and similar organizations. We also hope to 
generate long-term arrangements with these outside organizations through which students 
are regularly placed in routine programs of proven value to both the student and the 
organization. 

An expectation would also be for an individual in the organization to act as local 
supervisor for the student’s internship coursework credits. This would entail working 
with the student and an EDMS faculty member at the beginning of the experience to 
negotiate a project, hopefully part of the normal job activities of the student, that would 
result in a formal paper written by the student and approved by the other two individuals 
at the end of each semester. The intent is to provide an academic component over and 
above the student’s day-to-day job activities and to generate a document that can be used 
to evaluate the student’s overall learning. The paper will be deposited in the EDMS 
department office and will be the means to satisfy the “Scholarly Paper” requirement of 
the Graduate School for a Master of Arts degree. Some face-to-face meetings during the 
project between the student, the faculty member, and the supervisor would be arranged. 
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Masters-Level Programs for Assessment Professionals in Applied Settings 

All students would complete this core of courses (these are already taught on campus): 

Classroom Assessment (a measurement course for prospective teachers) 

Intro, to Research & Statistics (including research design and introductory stat.) 

Intro, to Analysis of Variance (analysis of variance designs through repeated measures) 
Measurement Theory (including classical and modem theories) 

Multiple Regression (including polynomial & logistic regression, & discrim, anal.) 

The two specializations are organized around unique competency areas. These are: 

1. Program in Assessment Development. Administration, and Scoring (requires at least 
two years of teaching experience for admission: requires nine credits of internship, 
spread over three semesters! 

Competency Area 1 : Writing both for style and for clarity - writing must be understood 
by test administrators and their supervisors, and by examinees so that all have the same 
opportunity to demonstrate their achievement (potential coursework: technical writing; 
internship) 

Competency Area 2: Understanding the impacts of deviations from standardized testing 
- ability to evaluate and recommend approval or disapproval of testing modifications for 
local conditions, such as accommodations for students with rare disabilities and 
unplanned events (potential coursework: classroom assessment; measurement theory; 
individual testing; assessment for special needs; internship) 

Competency Area 3: Knowledge of legal principles for testing and test security 
(potential coursework: classroom assessment; internship) 

Competency Area 4: Knowledge of ways to document how items and their scoring 
processes reflect domains to be assessed (potential coursework: classroom assessment; 
internship) 

Competency Area 5: Ability to develop and modify scoring materials for constructed- 
response items and to evaluate whether the items are aimed at the scoring materials 
(potential coursework: classroom assessment; internship) 

Competency Area 6: Knowledge of training techniques used, distribution methods for 
materials, ways to resolve concerns, and methods used for quality control, in scoring 
projects (potential coursework: internship) 

Competency Area 7: Ability to interpret and explain data used to evaluate the quality of 
items (potential coursework: classroom assessment; measurement theory; internship) 




An example internship for this specialization is with the Assessment Branch of the 
Maryland State Department of Education (MSDE). The student would spend two days 
per week at MSDE during the fall and spring semesters. The student would also spend 
the equivalent of a summer session at MSDE during the MSPAP scoring project. The 
student would participate in all phases of test development for MSPAP, beginning with 
task-writer training in the fall semester and ending with the MSPAP administration in the 
spring semester. An incumbent administration specialist and an incumbent scoring 
specialist at MSDE would share coordinated responsibilities as the student’s mentors. 
The student will be included at meetings and other relevant activities at MSDE, including 
the July MSPAP scoring project, and will perform job tasks under the supervision of the 
mentors. The student will receive nine credits for the internships, three credits for each 
of the fall semester, the spring semester, and the summer. The student will be required to 
turn in a log describing each day’s activities, insights learned from those activities, and 
questions raised that are explored during later activities. A brief evaluative report from 
the mentors will be forwarded to the professor directing the internship at the end of each 
three-credit experience. 

A sample schedule for this specialization is: 

Fall Semester: Classroom Assessment 

Introduction to Research & Statistics 

Technical Writing 

Internship 

Spring Semester: Measurement Theory 

Introduction to Analysis of Variance 
Assessment in Special Education 
Internship 

Summer Session: Multiple Regression 

Internship 
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2. Program in Data Analysis and Reporting (requires six credits of internship spread 
over two semesters') 

Competency Area 1 : Knowledge of and ability to explain the ways scores are processed 
to add value for interpretation (potential coursework: classroom assessment, 
measurement theory, introduction to statistics, introduction to program evaluation) 

Competency Area 2: Knowledge of and ability to explain fundamental univariate and 
bivariate statistics, and hypothesis testing including contingency tables, randomized and 
repeated-measures analysis of variance designs, and multiple and logistic regression 
(potential coursework: introduction to research, introduction to statistics, multiple 
regression) 

Competency Area 3: Ability to use data editing, database, spreadsheet, and statistical 
processing packages (potential coursework: short courses at the Computer Science 
Center; not-for-credit but can be listed on a resume) 

Competency Area 4: Ability to perform secondary data analyses that reflect sampling 
weights and design effects (potential coursework: large scale data analysis) 

Competency Area 5: Ability to set up and execute standard and specialized programs 
that implement applied statistics including exploratory and confirmatory factor analysis, 
structural equations modeling, and multilevel linear modeling (potential coursework: 
factor analysis; structural equations modeling; multilevel linear modeling) 

An example internship for this specialization is with the Results Branch of the Maryland 
State Department of Education. The student would spend two days per week for two 
semesters and would work with a data analysis specialist who would serve as the 
student’s mentor. The student would work on routine analyses at MSDE, analyses that 
are in response to special issues that arise, and analyses that are planned by the student 
for the purposes of the internship experience to implement at least one example of each 
of these techniques: multiple regression, logistic regression, two- or higher-way analysis 
of variance, exploratory factor analysis, confirmatory factor analysis, causal structural 
equations modeling, hierarchical linear modeling, and analysis of a large-scale database. 
The student will receive six credits for the two-semester experience. The student will be 
required to turn in a log describing each day’s activities, insights learned from those 
activities, and questions raised that are explored during later activities. A brief evaluative 
report from the mentor will be forwarded to the professor directing the internship at the 
end of each three-credit experience. 

This program includes a year-long structured sequence of courses as prerequisite for the 
higher-level analyses that must be studied before the student can profit from an 
internship. It is expected therefore that the student will be able to provide his or her own 
support for the first year. The prerequisite portion of the program may be taken part-time 
over a longer time frame, perhaps as a part-time student. It would not be unusual for 
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students to enter the program with some of these experiences in their undergraduate 
coursework. As a result, even though the entire program lists two courses more than the 
minimum required for a masters degree, it is unlikely that any one student would need to 
take all this coursework in his or her masters program. 



A sample schedule for this specialization is: 



Fall Semester: Classroom Assessment 

Introduction to Research & Statistics 

SPSS Short Course at the Computer Science Center 

SAS Short Course at the Computer Science Center 

Spring Semester: Introduction to Analysis of Variance 

Measurement Theory 

Database Short Course at the Computer Science Center 
Spreadsheet Short Course at the Computer Science Center 



Summer Session: Multiple Regression 

Large-Scale Database Analysis 

Fall Semester: Factor Analysis 

Multilevel Linear Models 
Internship 



Spring Semester: Structural Equations Modeling 

Introduction to Program Evaluation 
Internship 



Feasibility 

In our exploratory work, we have received almost universal interest on the part of outside 
agencies in receiving applications for employment from these programs’ graduates. On 
the other hand, many are reluctant to contribute their sites as placements for paid interns. 
An example: 

One question people may ask is: What benefit may sponsors get? What is the 
incentive for agencies to participate? It seems that you are asking them to 
contribute a lot — some funds and their time to educate the students. You 
may want to obtain some training grant from NSF, foundations, and possibly 
NAEP to support such a program. 

Of course, any program will need to be of mutual benefit to be successful. However, it is 
anticipated that the interns will have valuable skills that they will contribute to each 
sponsoring organization. While this may be of different degrees of utility in different 
organizations, it is the unusual site that has a perfect match between what it wants to do 
and the available staff. Since much of the intern’s time will be spent working on the 
normal projects of the organization as they are “learning by doing,” the productivity of 
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the mentors may actually increase instead of the organization losing capacity. Of course, 
it remains to be seen whether that is true in the organizations we are able to work with in 
the future. 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 

(Specific Document) 







TM030784 



I. DOCUMENT IDENTIFICATION: 



Tltle ’ Ap/>M AzteZSMeEt Pre>{e.i5,'^m(s Q-hri*- Mosf&rS 


L&/<o[ . 

r 


Author(s): L\) \ ( 


h'Qm £>■ 


Corporate Source: 




of Mafiylancf 


Publication Date: 



II. REPRODUCTION RELEASE: 



In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the 
monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy, 
and electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each document, and, if 
reproduction release is granted, one of the following notices is affixed to the document. 



If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign at the bottom 
of the page. 



The sample sticker shown below will be 
affixed to all Level 1 documents 


The sample sticker shown below will be 
affixed to all Level 2A documents 


The sample sticker shown below will be 
affixed to all Level 2B documents 


PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE, AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS ONLY, 
HAS BEEN GRANTED BY 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED BY 


A® 




JP 




JP 


c/ 








_ c/ 










TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 


1 




2A 




2B 


Level 1 


Level 2A 


Level 2B 



i t t 






Check here for Level 1 release, permitting 
reproduction and dissemination In microfiche or other 
ERIC archival media (e.g., electronic) end paper 
copy. 



Check here for Level 2A release, permitting Check here for Level 2B release, permitting 

reproduction and dissemination In microfiche and in reproduction and dissemination in microfiche only 

electronic media for ERIC archival collection 
subscribers only 



Documents will be processed as indicated provided reproduction quality permits. 

If permission to reproduce Is granted, but no box is checked, documents will be processed at Level 1. 



Sign 

here,-* 

mease 

ERIC 



1 hereby grant to the Educational Resources Information Center (ERIC) nonexclusive permission to reproduce and disseminate this document 
as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and its system 
contractors requires permission from the copyright holder. Exception is made for non-profit reproduction by libraries and other service agencies 


to satisfy information needs of educators in response to discrete inquiries. 


> 




Signature: % - , ) Q A 


Printed Name/Positionffitle: 


A <S<£,'ote. Pr o fe<£or 


LaJ xJLaj^ma iL^r 


(jJ i / fi to Sclvyfizfl' 


Organization/ Address: £^ 3 , College. *C 

Um.veJ-S' if o'f' 

Ce//eger«rt tfD ,3o74<2-lllS 


’"Srwafc as 


fa *30/'3/4 -VUS 


E-Mail Address: x , 

CPU 


Date - Cf! tlfaoDO 



(over) 



III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 



If permission to reproduce is not granted to ERIC. or. if you wish ERIC to cite the availability of the document from another source, please 
provide the following information regarding the availability of the document. (ERIC will not announce a document unless it is publicly 
available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more , 
stringent for documents that cannot be made available through EDRS.) 




IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 

Name: 

Address: 



V. WHERE TO SEND THIS FORM: 



Send this form to the following ERIC Clearinghouse: 

ERIC CLEARINGHOUSE ON ASSESSMENT AND 
UNIVERSITY OF MARYLAND 
1129 SHRIVERLAB 
COLLEGE PARK, MD 20772 
ATTN: ACQUISITIONS 



EVALUATION 




However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 



ERIC Processing and Reference Facility 

' 4483-A Forbes Boulevard 

Lanham, Maryland 20706 

Telephone: 301-552-4200 
Toll Free: 800-799-3742 
FAX: 301-552-4700 
e-mail: ericfac@ineted.gov 
WWW: http://ericfac.piccard.csc.com 



088 (Rev. 2/2000) 



